1. Field of the Invention
The present invention relates to a neural network which is widely utilized in various applications such as letter recognition, speech recognition, robot motion control, process control in general, and neuro-computers.
2. Description of the Background Art
The backward error propagation scheme is a very well known conventional mechanism for the learning of a multi-layer perceptron, for which much researches have been reported. The actual implementation of the neural network using this scheme has developed significantly. This backward error propagation scheme will be briefly outlined.
A conceptual configuration of a conventional neural network in a three-layer configuration is shown in FIG. 1, which generally comprises an input layer 1, an intermediate layer 2, and an output layer 3. In this neural network, a plurality of neurons are inter-connected through a plurality of synapse connections. In this configuration, learning is realized by comparing a teacher signal [t] with an output [O] outputted from the neuron of the output layer 3 in response to the input of an input pattern [X] entered from the neuron of the input layer 1.
For a certain neuron j, let an input of this neuron j be denoted as net.sub.pj, where a suffix .sub.p indicates that the input is the p-th input pattern [Xp]. Then, this net.sub.pj can be expressed by the following equation (1). ##EQU1## where .omega..sub.ij is a transmission efficiency of a synapse connection connecting the neuron i and the neuron j.
Next, let an input and output function of the neuron j be denoted as f.sub.j. Then, the output O.sub.pj of this neuron j in response to the p-th input pattern [Xp] can be expressed by the following equation (2). EQU O.sub.pj =f.sub.j (net.sub.pj) (2)
In this case, the evaluation value E.sub.p can be defined by the following equation (3). ##EQU2## where t.sub.pj is a desired output of the neuron j. In other words, this evaluation value E.sub.p reflects the difference between the actual output and the desired output, so that it is desirable to minimize this evaluation value.
In order to minimize this evaluation value E.sub.p of the equation (3) by using the steepest descent method, it is known that a partial derivative of E.sub.p with respect to .omega..sub.ij should be used as an update value .DELTA..sub.p .omega..sub.ij for the synapse transmission efficiency .omega..sub.ij. Consequently, it is known that the update value .DELTA..sub.p .omega..sub.ij for the synapse transmission efficiency can be expressed by the following equation (4). EQU .DELTA..sub.p .omega..sub.ij =.eta..delta..sub.pj O.sub.pj ( 4)
where .eta. is a relaxation coefficient and .delta..sub.pj is given by the following equation (5) when the neuron j belongs to the output layer 3, EQU .delta..sub.pj =(t.sub.pj -O.sub.pj)f.sub.j (net.sub.pj) (5)
and otherwise .delta..sub.pj is given by the following equation (6). ##EQU3## where ' denotes a derivative.
This manner of updating the synapse transmission efficiency according to the equation (4) has been known as the backward error propagation algorithm.
When this backward error propagation algorithm is applied to the neural network, as shown in FIG. 2, the input pattern [X] is entered into the neuron of the input layer 1, and propagated to the neuron of the intermediate layer 2 and the neuron of the output layer 3 through synapse connections, and the output [O] is outputted from the neuron of the output layer 3. On the other hand, the teacher signal [t] and this output [O] are compared at the neuron of the output layer 3 to obtain an error signal to be propagated through the neurons of each layer in a backward direction with respect to the propagation of the input, while updating the synapse transmission efficiency at each neuron according to the equation (4), so as to reduce the error of the output with respect to the teacher signal sequentially.
This process is called learning. Thus, in the backward error propagation scheme, it is necessary to provide a path for propagating the signal to be utilized in the learning process backwards, from the output side to the input side.
Now, this learning mechanism based on the backward error propagation scheme is known to work effectively in the computer simulation. However, the actual implementation of this learning mechanism by dedicated hardware has encountered various difficulties.
For example, in order to realize this learning mechanism based on the backward error propagation scheme in a large scale neural network by using a compact and inexpensive analog circuits, it becomes necessary to construct circuitry for carrying out the procedures described above by combining various circuit elements, such as multipliers and adders. However, in such a construction, the inevitable processing error introduced by the analog circuit for calculating the solution to the above equation (4) can severely deteriorate the performance of the neural network as a whole.
In addition, the backward error propagation scheme requires very high level precision in each neuron and each synapse for the backward propagation of the error signal, in order to achieve the effective learning, so that the very expensive, high precision circuit elements must be used for neurons and synapses, and even when such expensive circuit elements are used, there still remains a possibility for the performance of the neural network to be affected by offsets and variations among the circuit elements.
Moreover, there are cases in which it is quite difficult to obtain the truly optimum solution (absolute minimum) in the backward error propagation scheme using the steepest descent method.
Furthermore, the backward error propagation scheme is applicable only to the network having a multi-layer structure as shown in FIGS. 1 and 2. Consequently, there has been no known method for appropriately adjusting the synapse transmission efficiency for a case involving the feedback of the synapses or the loop formed by the synapses.